Many X Variables

Housing Price Models

Obviously, in reality, houses do not shrink as they age, but the model doesn’t know that! It only understands the world through the data and correlations it observes. In these data, older houses are also smaller, so it’s important to control for the size of a home to get a true measure of the effect of age on prices.

Another way to think about this is that you are trying to compare houses that are the same in every way except age. This would be like a Randomized Control Trial (RCT). “Controlling” for square footage gets us closer to mimicking an RCT by removing the effect of changes in square footage.

Housing Price Models

reg_beta <- lm(price ~ age + sqft, ames)
coef(reg_beta)

(Intercept)         age        sqft 
79973.60078 -1087.23673    95.96949 

1. \(\beta_0\)’s interpretation is now the price of a property where all \(X\) variables are equal to zero. In this case, the price of a house that has an age of zero and zero square footage. Of course, a house like this does not exist, so \(\beta_0\) is not very meaningful.
2. \(\beta_1\) is the effect of one additional year of age on price. The model has removed the effect of square footage from this coefficient since we have “controlled” for it by adding square footage into the model.¹
3. \(\beta_2\) is the effect of an additional square foot of size on final sale price, holding age constant.

This model also makes intuitive sense: it says that older houses are worth less and larger houses are worth more.

Housing Price Models

Let’s practice with another variable: bedrooms. Let’s estimate a model of the form:

1. We should expect that \(\gamma_1>0\) since bigger houses should be more valuable.
2. We should also expect \(\gamma_2>0\) since houses with more bedrooms should also be more valuable.

Housing Price Models

What does it mean to increase the number of bedrooms in a property while holding square footage constant? Suppose a home is 2,000 square feet with four rooms. Each room is 500 square feet (on average). If we add an additional room, each room would only be 400 square feet (on average). Therefore, adding an extra bedroom, while holding square footage constant, makes for a bunch of small rooms. This is not something that is typically sought after on the housing market, hence the negative coefficient.

Housing Price Models

Below is a progression of three models:

r1 <- lm(log(price) ~ log(sqft), ames)
r2 <- lm(log(price) ~ log(sqft) + bedrooms, ames)
r3 <- lm(log(price) ~ log(sqft) + bedrooms + age, ames)
regz <- list(`log(Price)` = r1,
             `log(Price)` = r2,
             `log(Price)` = r3)
coefz <- c("log(sqft)" = "log(Square Footage)",
           "bedrooms" = "Bedrooms",
           "age" = "Age")
gofz <- c("nobs", "r.squared")
library("modelsummary")
modelsummary(regz,
             title = "Determinants of Sale Price",
             estimate = "{estimate}{stars}",
             coef_map = coefz,
             gof_map = gofz)

Zillow

Imagine you are a data scientist for Zillow, and you can observe characteristics about homes for sale and the list price. You could estimate a model to explain list price with square footage, bedrooms, age, size of the backyard, whether the property has a pool, etc. Then, for each property, you could generate a fitted sale price. If the fitted price is above the list price, this represents an underpriced home for sale (and an investment opportunity). On the other hand, a fitted price below a list price represents an overpriced home.

Zillow

In 2018, Zillow started “Zillow Offers”, a division of their company dedicated to buying and selling homes based on pretty much the exact logic outlined above. However, Zillow failed in a major way.

Why? What happened? There were ultimately a lot of reasons for why this did not work, but a major reason was omitted variable bias. As an extreme, facetious example, consider 100 properties that are all identical. Of course, Zillow’s model will therefore predict the exact same sale price for each of these homes. However, the list prices of these homes will certainly not be the same. First, people are not robots and have different circumstances. This is going to introduce random noise into the list prices. Also, homeowners know a lot about their own homes compared to what Zillow can observe in data. This is going to introduce non-random noise into the list price.

Zillow

Zillow was betting on the random part outweighing the non-random part. For example, suppose Zillow identifies the five most under-priced homes out of the lot of one hundred. These homes could be under-priced because the homeowners need to move ASAP for one reason or another and are willing to sell below market value. Or, these homes could be under-priced because the home is infested with termites and smells like a sewer. Not that Zillow was buying a bunch of termite traps, but their models were unable to control for everything that impacts a price of a home and they were selecting homes that were under-priced for non-random reasons.

NFTs

Let’s use example data from a single NFT from 3/15/2021. By limiting it to only one NFT highlight, the prices are now completely determined by serial number. Below is a plot of serial number and list price.

This plot shows that most NFTs are relatively cheap with some very expensive ones. To get a clearer picture of the relationship, we can plot the data in a log-log format.

It would be unreasonable to purchase an NFT with serial number \(s\) for price \(p\) if you could purchase an NFT with a lower serial number \(s-\epsilon\) for the same price \(p\) (or lower). Remember, the lower the serial number, the higher the value! So, let’s only consider NFTs where we cannot get a lower serial number without paying more. These will be colored in red.¹

NFTs

The NFT that is farthest from the line is the circled one. This NFT’s serial number is 138 and was on sale for $494. The model wants to re-sell it for $599.

Of course, this sort of model is hypothetical and old news (the TopShot market has since crashed). However, the point I am trying to make is that Zillow omitted important variables in their models are the results were disastrous.

Omitted Variables

Hopefully by now, you’re convinced that omitted variable bias is a problem for prediction. Also, after seeing what happened in the age and square footage example, hopefully you can see that omitted variables can also bias our coefficient interpretation.

Unfortunately, sometimes we cannot observe every variable that we’d like to control for in our regression. However, we can sign the bias of our coefficients if we can anticipate correlation coefficients.

Consider the age and square footage example. Initially, our omitted variable was square footage. We knew that square footage was positively correlated with price but negatively correlated with age. So, when we put square footage into the model, the coefficient for age went from \(-1474.96\) to \(-1087.24\). So, the omitted variable bias was negative since it was pushing the coefficient down in the negative direction.

The same thing happened with square footage and bedrooms. Bedrooms and price are negatively correlated, but bedrooms and square footage are positively correlated. So, our coefficient moved from \(111.69\) to \(136.36\), which is the same direction as the previous coefficients. Therefore, omitting bedrooms also caused negative omitted variable bias in this regression.

Omitted Variables

If you have a model \(Y = \delta_0 + \delta_1\times A + \epsilon\) and \(B\) is your omitted variable, the bias in \(\delta_1\) will be:

Remember: positive (negative) bias means the magnitude of the coefficient is artificially pushed upwards (downwards).

Unimportant Variables

Now we know the effect of omitting important variables on both our predictions and coefficient interpretations. But what is the effect of the opposite: including unimportant variables?

Including unimportant variables will not bias your other coefficients because if they’re truly unimportant, we would expect \(\beta_i = 0\). Take a look at the above table – if \(B\) does not effect \(Y\) whatsoever, then the bias will be neither positive nor negative regardless of \(B\)’s correlation with \(A\).

If this is the case, why don’t we throw every single variable we can think of into the model?! Well, there’s no such thing as a free lunch. Each time we include a new variable, especially when the new variable is highly correlated with one or more other variables, we lose a little bit of our precision. In other words, including new variables generally increases the standard errors of the coefficients for the other variables in the model. This makes it more difficult to effectively test our hypotheses.

Multicollinearity

Intuitively, adding two variables into a model that are highly correlated supplies the model with overlapping and redundant information. This makes it difficult for the model to parse out the effect of each variable individually. The problem of redundant information in a model is called multicollinearity. If two variables contain exactly the same information, the regression cannot differentiate between the two variables and therefore cannot be estimated without removing one of the variables. This is called perfect multicollinearity.

Multicollinearity

Multicollinearity occurs for one of two reasons:

1. Structural: Sometimes we create variables based on already existing variables. For example, we created an “age” variable from the “year built” variable. Year built and age are highly correlated. In fact, since one is a linear function of the other¹, including both into a regression will create perfect multicollinearity. Put differently, since the correlation between the two is -1, they contain exactly the same information.

lm(price ~ age + yr_built, ames)

Call:
lm(formula = price ~ age + yr_built, data = ames)

Coefficients:
(Intercept)          age     yr_built  
     239269        -1475           NA  

2. Data-Based: Other times, variables are correlated simply because we are working with observational data. For example, in reality, the number of hours of SAT tutoring for a student is highly correlated with their family income. If we had an RCT that randomized the amount of tutoring students got, we wouldn’t have to worry about parental income. However, in an observational setting, this is just a fact of the matter.

Multicollinearity

To demonstrate the effects of multicollinearity on model output, we are going to simulate some data. Basically, we are going to simulate the following data generating process:

The first time we run the model, \(X_1\) and \(X_2\) will be completely uncorrelated. However, we are going to use a for loop to re-run the model over and over, increasing the correlation between the two variables after each iteration. The code is not necessarily important, but you can still hopefully follow along.

set.seed(757)
N <- 100 # sample size
x1 <- rnorm(N, 0, 1) # create X1
e <-  rnorm(N, 0, 3) # create unobserved error
all1 <- list()
for(a in seq(0, .99, 0.01)){
  
  # X2 is a combination of randomness and X1
  # When a = 0, X2 and X1 are completely uncorrelated
  x2 <- ((1-a)*rnorm(N, 0, 1)) + (a*x1)
  rho <- cor(x1, x2) # calculate correlation
  y <- x1 + x2 + e # Data Generating Process
  # Estimate regression and store coefficients
  tmp <- as.data.frame(coef(summary(lm(y ~ x1 + x2))))
  # Store correlation
  tmp$a <- rho
  all1[[length(all1) + 1]] <- tmp # save results
}

# Combine results
all <- do.call(rbind, all1)
# Round off correlation
all$a <- round(all$a, 2)

layout(matrix(c(1,1,2,3), 2, 2, byrow = TRUE))
par(mar = c(4.1, 4.1, 1.1, 1.1))
plot(all$a[c(T,F,F)], las = 1, pch = 19,
     col = scales::alpha("black", 0.6), cex = 1.25,
     xlab = "Loop Iteration Number",
     ylab = "Corr. Between X1 and X2")
abline(h = c(0, 0.5, 0.9), lty = 2)
plot(all$a[c(F,T,F)], all$Estimate[c(F,T,F)], las = 1,
     xlab = "Correlation", cex = 1.25,
     ylab = "Coefficient Estimate", pch = 19,
     ylim = range(all$Estimate[c(F,T,F)][-which(all$Estimate[c(F,T,F)] == max(all$Estimate[c(F,T,F)]))]),
     col = scales::alpha("black", 0.6))
abline(h = 1)
plot(all$a[c(F,T,F)], all$`t value`[c(F,T,F)], las = 1,
     xlab = "Correlation", cex = 1.25,
     ylab = "t-Statistic", pch = 19,
     col = scales::alpha("black", 0.6))
abline(h = 1.96, lty = 2)
layout(matrix(c(1), 1, 1, byrow = TRUE))

Multicollinearity

Examining these three figures:

1. As expected, the first few iterations have a very low correlation coefficient, hovering between 0 and 0.2. Then, the correlation coefficient reaches 0.5 by around iteration 40. Finally, the correlation reaches 0.9 by iteration 70 or so.
2. Up until the correlation reaches $$0.95, the coefficient estimate is extremely stable and hovers around 1.
3. However, the t-statistic for the coefficients signals that the regression is unable to differentiate between the coefficient estimate (which is about 1) and 0 at the 95% level starting when the correlation is around 0.5.

All three of these plots provide evidence that multicollinearity does not bias coefficients (until it becomes extreme), but does hinder the precision of the estimates.

Choosing Variables

To summarize the effect of including (or not including) additional regressors:

1. Omitting important variables biases coefficient estimates and model predictions.
2. Including unimportant variables reduces the precision of the coefficient estimates.

The way you should think about adding new variables to your model is through a bias-variance tradeoff. We know that coefficients can be biased if we omit important variables, but our standard errors get larger as the number of parameters we have to estimate increases.

It’s important to consider only the most important variables in your model to minimize both bias and variance. You might ask: how can I tell which variables are most important? To that I would say: good question, and the best answer is economic theory.

Choosing Variables

Economic theory should guide your choices about which variables to include/exclude and which functional forms you should use (log(), x + x^2, etc.). Unfortunately, this might seem like an unsatisfying answer, but econometrics is as much an art as it is a science. As you become more comfortable with econometrics, selecting variables and functional forms will become more natural.

In the next portion of the module, we will discuss how to incorporate binary/categorical variables into our models.